Papers with Semantic Textual Similarity
SEAVER: Attention Reallocation for Mitigating Distractions in Language Models for Conditional Semantic Textual Similarity Measurement (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Conditional Semantic Textual Similarity (C-STS) introduces specific limiting conditions to the traditional Semantics task. |
| Approach: | They propose a conditional semantic textual similarity (C-STS) task that introduces specific limiting conditions to the traditional Semantic Textual Similarity task. |
| Outcome: | The proposed model outperforms existing models on the C-STS-2023 test set and consistently improves on million-scale fine-tuning baseline models (up to 3 points). |
KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work on multimodal sentence embeddings took negative samples without reviewing, resulting in noisy and noisy negative samples. |
| Approach: | They propose a multimodal contrastive learning approach that inherits the knowledge from the teacher model to learn the difference between positive and negative instances. |
| Outcome: | The proposed approach can detect noisy and wrong negative samples before they are calculated in the contrastive objective. |
Capturing the Relationship Between Sentence Triplets for LLM and Human-Generated Texts to Enhance Sentence Embeddings (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in building sentence embedding models have centered on replacing traditional human-generated text datasets with those generated by LLMs. |
| Approach: | They propose a loss function that incorporates Positive-Negative sample Augmentation within the contrastive learning objective to enhance sentence embeddings using both human and LLM-generated datasets. |
| Outcome: | The proposed model mitigates the sentence anisotropy problem in Wikipedia corpus and improves Spearman’s correlation in standard Semantic Textual Similarity (STS) tasks (+1.47% compared to CLHAIF). |
Going Beyond Sentence Embeddings: A Token-Level Matching Algorithm for Calculating Semantic Textual Similarity (2023.acl-short)
Copied to clipboard
| Challenge: | Semantic Textual Similarity (STS) measures the degree to which the underlying semantics of paired sentences are equivalent. |
| Approach: | They propose a token-level matching inference algorithm which can be applied on top of any language model to improve its performance on STS task. |
| Outcome: | The proposed method improves the performance of almost all language models, with up to 12.7% gain in Spearman’s correlation. |
Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for extracting conditional text embeddings from large language models (LLMs) relying on prompts often fails to produce high-quality conditional embeddables, resulting in degradation of quality. |
| Approach: | They propose a plug-and-play method that constructs unconditional general text embeddings and uses them to refine conditional text embeds. |
| Outcome: | The proposed method improves performance of prompt-based methods on clustering, Semantic Textual Similarity, and triplet alignment datasets. |
Using Semantic Similarity as Reward for Reinforcement Learning in Sentence Generation (P19-2)
Copied to clipboard
| Challenge: | Existing models for sentence generation use cross-entropy loss as the loss function . however, cross-etropy is unable to evaluate sentences as a whole and lacks flexibility . et al., 2018: a novel approach to improve sentence generation models . |
| Approach: | They propose a method to train a model using estimated semantic similarity between output and reference sentences to alleviate cross-entropy loss problems. |
| Outcome: | The proposed model improves the BLEU scores from the baseline LSTM NMT model. |
TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to learn sentence embeddings require labeled data, but it is expensive. |
| Approach: | They propose an unsupervised method which learns sentence embeddings using unlabeled data . they propose a transformer-based sequence denoising auto-encoder which can be used for training . |
| Outcome: | The proposed method outperforms existing methods on four datasets from heterogeneous domains. |
Compositional Evaluation on Japanese Textual Entailment and Similarity (2022.tacl-1)
Copied to clipboard
| Challenge: | Despite growing interest in linguistic universals, most NLI/STS studies focus on English. |
| Approach: | They propose a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. |
| Outcome: | The proposed datasets show that pre-trained language models are insensitive to word order and case particles. |
From Semantics to Style: A Cross-Dataset Comparative Framework for Sentence Similarity Predictions (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing frameworks for analyzing text embedding models are limited. |
| Approach: | They propose a framework that uses lightweight poolers to analyze STS, PI, and Triplet datasets. |
| Outcome: | The proposed framework shows that the model captures semantic differences between sentences and is consistent across datasets. |
Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on prompt engineering to extract sentence embeddings from large language models (LLMs) but these models are mostly decoder-only and the earlier tokens in the sentence cannot attend to the latter, resulting in biased encoding of sentence information and cascading effects on the final decoded token. |
| Approach: | They propose a plug-and-play and training-free technique that prepends each layer’s decoded sentence embedding to the beginning of the sentence in the next layer’ s input. |
| Outcome: | The proposed technique can significantly improve the performance of existing prompt-based sentence embedding methods across different LLMs while incurring negligible additional inference cost. |
Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on prompt engineering to encode the full semantics of a sentence into the embedding of the last token. |
| Approach: | They propose a technique that introduces an extra auxiliary prompt to elicit better sentence embedding . they propose to use the hidden state of the token as the sentence embedded in LLMs . |
| Outcome: | The proposed technique can improve performance of existing prompt-based methods on STS tasks and downstream classification tasks. |
Alleviating Over-smoothing for Unsupervised Sentence Representation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to learn better unsupervised sentence representations have been successful . over-smoothing problem in unsupervised sentences reduces the capacity of powerful PLMs . |
| Approach: | They propose a method to solve the over-smoothing problem in unsupervised sentence representations by combining negatives from PLMs intermediate layers. |
| Outcome: | The proposed method improves on different strong baselines on Semantic Textual Similarity and Transfer datasets. |
NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents (2024.acl-long)
Copied to clipboard
| Challenge: | (large) language models struggle to process long sequences due to the quadratic scaling of the underlying attention mechanism. |
| Approach: | They propose a Masked Language Model operating on higher-level semantic representations in the form of text embeddings to solve this problem. |
| Outcome: | The proposed model outperforms larger embedding models on three types of tasks. |
GASE: Generatively Augmented Sentence Encoding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Generatively Augmented Sentence Encoding variates the input text by paraphrasing, summarizing, or extracting keywords, followed by pooling the original and synthetic embeddings. |
| Approach: | They propose a training-free approach to improve sentence embeddings by applying generative text models for data augmentation at inference time. |
| Outcome: | The proposed approach does not require access to model parameters or computational resources typically required for fine-tuning state-of-the-art models. |
Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan (2021.findings-acl)
Copied to clipboard
Jordi Armengol-Estapé, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, Marta Villegas
| Challenge: | Multilingual language models have been a crucial breakthrough for under-resourced languages . however, the superiority of language-specific models has already been proven for underresourced ones . |
| Approach: | They propose to build a monolingual monolingual model that is comparable to state-of-the-art large multilingual models. |
| Outcome: | The proposed model consistently outperforms state-of-the-art models across tasks and settings. |
Meta-Task Prompting Elicits Embeddings from Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for large language modeling are based on task-related instructions or prompts. |
| Approach: | They propose a method for generating high-quality sentence embeddings from Large Language Models (LLMs) using meta-task prompts. |
| Outcome: | The proposed method produces high-quality sentences without fine-tuning . it excels on STS benchmarks and in downstream tasks, surpassing models with similar prompts . |
Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss (2024.emnlp-main)
Copied to clipboard
| Challenge: | despite its efficiency, Sentence-BERT ignores the progressive nature of semantic relationships, despite a promising approach . contrastive learning methods have improved performance on renowned STS benchmarks, but they fail to leverage fine-grained information. |
| Approach: | They propose a regression framework that categorizes text pairs as either semantically similar or dissimilar . they propose two loss functions: Translated ReLU and Smooth K2 Loss to bridge this gap . |
| Outcome: | The proposed method achieves convincing performance across seven established STS benchmarks. |
Contextualized Semantic Distance between Highly Overlapped Texts (2023.findings-acl)
Copied to clipboard
| Challenge: | Conventional semantic metrics are based on word representations and are vulnerable to disturbance of overlapped components with similar representations. |
| Approach: | They propose a mask-and-predict strategy to evaluate the semantic distance between the overlapped sentences using words in the longest common sequence as neighboring words and use masked language modeling to predict their positions. |
| Outcome: | The proposed method outperforms the state-of-the-art in domain adaption by a huge margin. |
Exploiting Twitter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Semantic sentence embeddings are usually supervisedly built minimizing distances between pairs of embeddable sentences labelled as semantically similar by annotators. |
| Approach: | They propose a language-independent approach to build large datasets of pairs of informal texts weakly similar, without manual human effort, exploiting Twitter’s powerful signals of relatedness: replies and quotes of tweets. |
| Outcome: | The proposed model learns classical Semantic Textual Similarity, and excels on tasks where pairs of sentences are not exact paraphrases. |
Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similarity (2024.emnlp-main)
Copied to clipboard
| Challenge: | Semantic Textual Similarity (STS) is a key indicator of the encoding capabilities of embedding models. |
| Approach: | They propose to use Pearson’s correlation coefficient as a loss function to refine model performance beyond contrastive learning to achieve a Spearman’s ceiling. |
| Outcome: | The proposed method surpasses state-of-the-art strategies with minimal amount of fine-grained annotated samples. |
Representational Isomorphism and Alignment of Multilingual Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing isomorphism of sentence representations can facilitate representational alignments in zero-shot and few-shot settings. |
| Approach: | They propose to apply a contrastive objective to LLMs with a small number of translation pairs to improve models' performance on Semantic Textual Similarity tasks. |
| Outcome: | The proposed representation-level approach significantly improves on Semantic Textual Similarity (STS) tasks across languages even without a monolingual objective. |
Text Representation Distillation via Information Bottleneck Principle (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have recently shown great success in text representation field, however, the high computational cost and high-dimensional representation of PLMs pose significant challenges for practical applications. |
| Approach: | They propose a Knowledge Distillation method that distills large models into smaller representation models to reduce performance degradation after distillation. |
| Outcome: | Empirical results on two main downstream applications of the proposed method show that it reduces the risk of over-fitting and maximizes the mutual information between the model and the input data. |
Beyond Averages: Learning with Annotator Disagreement in STS (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to capture and model disagreement in Semantic Textual Similarity (STS) ignore label dispersion and incentivize models to ignore uncertainty crucial for practical settings. |
| Approach: | They propose to capture and model disagreement in Semantic Textual Similarity (STS) a lightweight truncated Gaussian head and a cross-encoder are used to model disagreement . |
| Outcome: | The proposed approach improves accuracy and calibration of models to human judgments. |